{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Rent Listing Inqueries Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rental Listing Inquiries数据集是Kaggle平台上的一个分类竞赛任务，需要根据公寓的特征来预测其受欢迎程度（用户感兴趣程度分为高、中、低三类）。其中房屋的特征x共有14维，响应值y为用户对该公寓的感兴趣程度。评价标准为logloss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6.参数调优：reg_alpha & reg_lambda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用GridSearchCV进行两个参数的同时调优  \n",
    "注：GridSearchCV中最终判定的是scoring越大的模型越好，若用log_loss作为评价指标，在函数中要用neg_log_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>...</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "      <th>interest_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.5</td>\n",
       "      <td>3</td>\n",
       "      <td>3000</td>\n",
       "      <td>1200.0</td>\n",
       "      <td>750.000000</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>4.5</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>5465</td>\n",
       "      <td>2732.5</td>\n",
       "      <td>1821.666667</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.0</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3275</td>\n",
       "      <td>1637.5</td>\n",
       "      <td>1637.500000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>18</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3350</td>\n",
       "      <td>1675.0</td>\n",
       "      <td>670.000000</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>28</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 228 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  price  price_bathrooms  price_bedrooms  room_diff  \\\n",
       "0        1.5         3   3000           1200.0      750.000000       -1.5   \n",
       "1        1.0         2   5465           2732.5     1821.666667       -1.0   \n",
       "2        1.0         1   2850           1425.0     1425.000000        0.0   \n",
       "3        1.0         1   3275           1637.5     1637.500000        0.0   \n",
       "4        1.0         4   3350           1675.0      670.000000       -3.0   \n",
       "\n",
       "   room_num  Year  Month  Day       ...        walk  walls  war  washer  \\\n",
       "0       4.5  2016      6   24       ...           0      0    0       0   \n",
       "1       3.0  2016      6   12       ...           0      0    0       0   \n",
       "2       2.0  2016      4   17       ...           0      0    0       0   \n",
       "3       2.0  2016      4   18       ...           0      0    0       0   \n",
       "4       5.0  2016      4   28       ...           0      0    1       0   \n",
       "\n",
       "   water  wheelchair  wifi  windows  work  interest_level  \n",
       "0      0           0     0        0     0               1  \n",
       "1      0           0     0        0     0               2  \n",
       "2      0           0     0        0     0               0  \n",
       "3      0           0     0        0     0               2  \n",
       "4      0           0     0        0     0               2  \n",
       "\n",
       "[5 rows x 228 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "train = pd.read_csv(dpath +\"RentListingInquries_FE_train.csv\")\n",
    "test = pd.read_csv(dpath +\"RentListingInquries_FE_test.csv\")\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>...</th>\n",
       "      <th>virtual</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2950</td>\n",
       "      <td>1475.000000</td>\n",
       "      <td>1475.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>950.000000</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3758</td>\n",
       "      <td>1879.000000</td>\n",
       "      <td>1879.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3300</td>\n",
       "      <td>1650.000000</td>\n",
       "      <td>1100.000000</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>4900</td>\n",
       "      <td>1633.333333</td>\n",
       "      <td>1633.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 227 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  price  price_bathrooms  price_bedrooms  room_diff  \\\n",
       "0        1.0         1   2950      1475.000000     1475.000000        0.0   \n",
       "1        1.0         2   2850      1425.000000      950.000000       -1.0   \n",
       "2        1.0         1   3758      1879.000000     1879.000000        0.0   \n",
       "3        1.0         2   3300      1650.000000     1100.000000       -1.0   \n",
       "4        2.0         2   4900      1633.333333     1633.333333        0.0   \n",
       "\n",
       "   room_num  Year  Month  Day  ...   virtual  walk  walls  war  washer  water  \\\n",
       "0       2.0  2016      6   11  ...         0     0      0    0       0      0   \n",
       "1       3.0  2016      6   24  ...         0     0      0    1       0      0   \n",
       "2       2.0  2016      6    3  ...         0     0      0    0       0      0   \n",
       "3       3.0  2016      6   11  ...         0     0      0    0       0      0   \n",
       "4       4.0  2016      4   12  ...         0     0      0    1       0      0   \n",
       "\n",
       "   wheelchair  wifi  windows  work  \n",
       "0           0     0        0     0  \n",
       "1           0     0        0     0  \n",
       "2           0     0        0     0  \n",
       "3           1     0        0     0  \n",
       "4           0     0        0     0  \n",
       "\n",
       "[5 rows x 227 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "发现测试集中没有y，这次作业无法得到在测试集上的表现情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*********** train **********\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 49352 entries, 0 to 49351\n",
      "Columns: 228 entries, bathrooms to interest_level\n",
      "dtypes: float64(9), int64(219)\n",
      "memory usage: 85.8 MB\n",
      "************ test ***********\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 74659 entries, 0 to 74658\n",
      "Columns: 227 entries, bathrooms to work\n",
      "dtypes: float64(9), int64(218)\n",
      "memory usage: 129.3 MB\n"
     ]
    }
   ],
   "source": [
    "print(\"*********** train **********\")\n",
    "train.info()\n",
    "print(\"************ test ***********\")\n",
    "test.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试集中少一维，即y标签"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAELCAYAAAARNxsIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGx5JREFUeJzt3X2wHXWd5/H3h/AgPibIhcokYYKaGUUdo14hypSj6EDAHRMpUChHIkNtxAKF0rUEd5aMPMyM4ygrM8pupogkrkOIiBI1GDMMyII8JGB4CJHlioxcyZJgeNQVivDZP/p35eTm5N7OpU9OTu7nVdV1ur/96z7fwy340v379a9lm4iIiCbs0e0EIiJi95GiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIas2e3E9jZ9t9/f0+fPr3baURE9JTbbrvtEdt9o7Ubd0Vl+vTprFmzpttpRET0FEn/Uaddbn9FRERjOlZUJL1I0q2S7pC0TtLnS/xSSb+QtLYsM0tcki6SNCDpTklvaTnXPEn3lWVeS/ytku4qx1wkSZ36PRERMbpO3v56GjjC9lOS9gJukHR12fcZ21cMa380MKMshwEXA4dJ2g9YAPQDBm6TtNz2o6XNfOBmYAUwG7iaiIjoio5dqbjyVNncqywjzbM/B1hSjrsZmChpMnAUsMr25lJIVgGzy76X277J1fz9S4C5nfo9ERExuo72qUiaIGktsJGqMNxSdl1QbnFdKGmfEpsCPNhy+GCJjRQfbBNvl8d8SWskrdm0adML/l0REdFeR4uK7S22ZwJTgUMlvQE4G3gt8DZgP+CzpXm7/hCPId4uj4W2+2339/WNOiIuIiLGaKeM/rL9GHAdMNv2hnKL62ng68ChpdkgMK3lsKnAQ6PEp7aJR0REl3Ry9FefpIllfV/gvcDPSl8IZaTWXODucshy4KQyCmwW8LjtDcBK4EhJkyRNAo4EVpZ9T0qaVc51EnBVp35PRESMrpOjvyYDiyVNoCpey2x/X9K/S+qjun21Fji1tF8BHAMMAL8FTgawvVnSecDq0u5c25vL+seBS4F9qUZ9ZeRXREQXqRo4NX709/c7T9RH7NoO/6fDu53Cbu/GT9y4Q+0l3Wa7f7R2eaI+IiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYmIiMakqERERGNSVCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMR0rKpJeJOlWSXdIWifp8yV+sKRbJN0n6XJJe5f4PmV7oOyf3nKus0v8XklHtcRnl9iApLM69VsiIqKeTl6pPA0cYftNwExgtqRZwBeAC23PAB4FTintTwEetf0a4MLSDkmHACcArwdmA1+TNEHSBOCrwNHAIcCJpW1ERHRJx4qKK0+Vzb3KYuAI4IoSXwzMLetzyjZl/3skqcSX2n7a9i+AAeDQsgzYvt/2M8DS0jYiIrqko30q5YpiLbARWAX8HHjM9rOlySAwpaxPAR4EKPsfB17ZGh92zPbiERHRJR0tKra32J4JTKW6snhdu2blU9vZt6PxbUiaL2mNpDWbNm0aPfGIiBiTnTL6y/ZjwHXALGCipD3LrqnAQ2V9EJgGUPa/AtjcGh92zPbi7b5/oe1+2/19fX1N/KSIiGijk6O/+iRNLOv7Au8F1gPXAseVZvOAq8r68rJN2f/vtl3iJ5TRYQcDM4BbgdXAjDKabG+qzvzlnfo9ERExuj1HbzJmk4HFZZTWHsAy29+XdA+wVNL5wE+BS0r7S4BvSBqgukI5AcD2OknLgHuAZ4HTbG8BkHQ6sBKYACyyva6DvyciIkbRsaJi+07gzW3i91P1rwyP/w44fjvnugC4oE18BbDiBScbERGNyBP1ERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGjMqEVF0ksk7VHW/0jS+yXt1fnUIiKi19S5UrkeeJGkKcA1wMnApZ1MKiIielOdoiLbvwWOBf7J9geo3gkfERGxlVpFRdLbgQ8DPyixTk6ZHxERPapOUTkTOBv4Tnm3yauoXrQVERGxlVGvOGz/GPixpJeU7fuBT3Y6sYiI6D11Rn+9vbytcX3ZfpOkr3U8s4iI6Dl1bn/9d+Ao4NcAtu8A3tnJpCIiojfVevjR9oPDQls6kEtERPS4OqO4HpT0DsCS9qbqT1nf2bQiIqIX1blSORU4DZgCDAIzy3ZERMRWRi0qth+x/WHbB9o+wPZf2v71aMdJmibpWknrJa2TdEaJ/42kX0laW5ZjWo45W9KApHslHdUSn11iA5LOaokfLOkWSfdJurxcSUVERJfUGf21WNLElu1JkhbVOPezwKdtvw6YBZwmaehJ/AttzyzLinLeQ4ATgNcDs4GvSZogaQLwVeBoqif5T2w5zxfKuWYAjwKn1MgrIiI6pM7trz+x/djQhu1HgTePdpDtDbZvL+tPUvXDTBnhkDnAUttP2/4FMAAcWpYB2/fbfgZYCsyRJOAI4Ipy/GJgbo3fExERHVKnqOwhadLQhqT92MFpWiRNpypEt5TQ6ZLulLSo5dxTgNZRZoMltr34K4HHbD87LB4REV1Sp6h8CfiJpPMknQf8BPiHul8g6aXAt4EzbT8BXAy8mqrDf0M5P4DaHO4xxNvlMF/SGklrNm3aVDf1iIjYQXU66pcAxwEPAxuBY21/o87Jy3tXvg180/aV5XwP295i+zngX6hub0F1pTGt5fCpwEMjxB8BJkrac1i83W9YaLvfdn9fX1+d1CMiYgzqvvnxZ8CVwFXAU5IOGu2A0udxCbDe9pdb4pNbmn0AuLusLwdOkLSPpIOBGcCtwGpgRhnptTdVZ/5y26aa2PK4cvy8kl9ERHTJqH0jkj4BLKC6UtlCddvJwJ+McujhwEeAuyStLbHPUY3emlnO8QDwMYAyA/Iy4B6qkWOn2d5ScjgdWAlMABbZXlfO91lgqaTzgZ9SFbGIiOiSOh3uZwB/XOfZlFa2b6B9v8eKEY65ALigTXxFu+PKjMmHDo9HRER31Ln99SDweKcTiYiI3lfnSuV+4DpJPwCeHgq29pNERERAvaLyy7LsXZaIiIi26rz58fMAkl5i+zedTykiInpV3vwYERGNyZsfIyKiMXnzY0RENCZvfoyIiMbkzY8REdGYEa9UyguyPmL7wzspn4iI6GEjXqmUubfm7KRcIiKix9XpU7lR0j8DlwO/f05l6K2OERERQ+oUlXeUz3NbYqZ6lW9ERMTvjdansgdwse1lOymfiIjoYaP1qTwHnL6TcomIiB5XZ0jxKkn/RdI0SfsNLR3PLCIiek6dPpW/Kp+tz6YYeFXz6URERC+rM0vxwTsjkYiI6H113lF/Uru47SXNpxMREb2szu2vt7Wsvwh4D3A7kKISERFbqXP76xOt25JeAXyjYxlFRETPqjX1/TC/BWaM1qiMFrtW0npJ6ySdUeL7SVol6b7yOanEJekiSQOS7pT0lpZzzSvt75M0ryX+Vkl3lWMukqQx/J6IiGhInTc/fk/S8rJ8H7gXuKrGuZ8FPm37dcAs4DRJhwBnAdfYngFcU7YBjqYqVjOA+cDF5fv3AxYAhwGHAguGClFpM7/luNk18oqIiA6p06fyjy3rzwL/YXtwtINsbwA2lPUnJa2nmj5/DvCu0mwxcB3w2RJfYtvAzZImSppc2q6yvRlA0ipgtqTrgJfbvqnElwBzgatr/KaIiOiAOkXll8AG278DkLSvpOm2H6j7JZKmA28GbgEOLAUH2xskHVCaTQFa3zA5WGIjxQfbxCMiokvq9Kl8C3iuZXtLidUi6aXAt4EzbT8xUtM2MY8h3i6H+ZLWSFqzadOm0VKOiIgxqlNU9rT9zNBGWd+7zskl7UVVUL5p+8oSfrjc1qJ8bizxQWBay+FTgYdGiU9tE9+G7YW2+2339/X11Uk9IiLGoE5R2STp/UMbkuYAj4x2UBmJdQmw3vaXW3YtB4ZGcM3j+U7/5cBJZRTYLODxcptsJXCkpEmlg/5IYGXZ96SkWeW7TqLeAIKIiOiQOn0qpwLfLC/qguoKoe1T9sMcDnwEuEvS2hL7HPD3wDJJp1D11xxf9q0AjgEGqIYtnwxge7Ok84DVpd25Q532wMeBS4F9qTro00kfEdFFdR5+/Dkwq/SNyPaTdU5s+wba93tA9VT+8PZm60krW/ctAha1ia8B3lAnn4iI6Lw6z6n8raSJtp8qQ4MnSTp/ZyQXERG9pU6fytG2HxvasP0o1W2qiIiIrdQpKhMk7TO0IWlfYJ8R2kdExDhVp6P+fwHXSPo61XMgf0X1JHxERMRW6nTU/4OkO4H3ltB5tld2Nq2IiOhFda5UAH4K7EV1pfLTzqUTERG9rM7orw8CtwLHAR8EbpF0XKcTi4iI3lPnSuW/Am+zvRFAUh/wb8AVnUwsIiJ6T53RX3sMFZTi1zWPi4iIcabOlcoPJa0ELivbH6KaUiUiImIrdUZ/fUbSscCfUk27stD2dzqeWURE9Jxao7/KtPVXjtowIiLGtfSNREREY1JUIiKiMdstKpKuKZ9f2HnpRERELxupT2WypD8D3i9pKcPejWL79o5mFhERPWekonIOcBbVu9+/PGyfgSM6lVRERPSm7RYV21cAV0j6b7bP24k5RUREj6rznMp5kt4PvLOErrP9/c6mFRERvajOhJJ/B5wB3FOWM0osIiJiK3UefnwfMNP2cwCSFlNNf392JxOLiIjeU/c5lYkt66/oRCIREdH76hSVvwN+KunScpVyG/C3ox0kaZGkjZLubon9jaRfSVpblmNa9p0taUDSvZKOaonPLrEBSWe1xA+WdIuk+yRdLmnvuj86IiI6Y9SiYvsyYBbV3F9XAm+3vbTGuS8FZreJX2h7ZllWAEg6BDgBeH055muSJkiaAHwVOBo4BDixtAX4QjnXDOBR4JQaOUVERAfVuv1le4Pt5bavsv1/ax5zPbC5Zh5zgKW2n7b9C2AAOLQsA7bvt/0MsBSYI0lUz8kMvShsMTC35ndFRESHdGPur9Ml3Vluj00qsSnAgy1tBktse/FXAo/ZfnZYvC1J8yWtkbRm06ZNTf2OiIgYZmcXlYuBVwMzgQ3Al0pcbdp6DPG2bC+03W+7v6+vb8cyjoiI2kYsKpL2aO1of6FsP2x7Sxme/C9Ut7egutKY1tJ0KvDQCPFHgImS9hwWj4iILhqxqJT/+N8h6aAmvkzS5JbNDwBDBWs5cIKkfSQdDMwAbgVWAzPKSK+9qTrzl9s2cC1wXDl+HnBVEzlGRMTY1Xn4cTKwTtKtwG+GgrbfP9JBki4D3gXsL2kQWAC8S9JMqltVDwAfK+daJ2kZ1RP7zwKn2d5SznM6sBKYACyyva58xWeBpZLOp3oY85I6PzgiIjqnTlH5/FhObPvENuHt/off9gXABW3iK4AVbeL38/zts4iI2AXUmVDyx5L+EJhh+98kvZjqqiEiImIrdSaU/M9Uz4P8zxKaAny3k0lFRERvqjOk+DTgcOAJANv3AQd0MqmIiOhNdYrK0+VpdgDKMN7tPhMSERHjV52i8mNJnwP2lfTnwLeA73U2rYiI6EV1ispZwCbgLqohwCuAv+5kUhER0ZvqjP56rkx5fwvVba97y8OHERERWxm1qEh6H/A/gJ9Tzbl1sKSP2b6608lFRERvqfPw45eAd9seAJD0auAHQIpKRERspU6fysahglLcD2zsUD4REdHDtnulIunYsrpO0gpgGVWfyvFUEz1GRERsZaTbX3/Rsv4w8GdlfRMwadvmEREx3m23qNg+eWcmEhERva/O6K+DgU8A01vbjzb1fUREjD91Rn99l2rK+u8Bz3U2nYiI6GV1isrvbF/U8UwiIqLn1SkqX5G0APgR8PRQ0PbtHcsqIiJ6Up2i8kbgI8ARPH/7y2U7Ypf0y3Pf2O0UxoWDzrmr2ynELqZOUfkA8KrW6e8jIiLaqfNE/R3AxE4nEhERva/OlcqBwM8krWbrPpUMKY6IiK3UKSoLxnJiSYuA/0Q1d9gbSmw/4HKqZ14eAD5o+1FJAr4CHAP8Fvjo0EAASfN4/v0t59teXOJvBS4F9qV6x8sZmZI/IqK7Rr39ZfvH7ZYa574UmD0sdhZwje0ZwDVlG+BoYEZZ5gMXw++L0ALgMOBQYIGkoSliLi5th44b/l0REbGTjVpUJD0p6Ymy/E7SFklPjHac7euBzcPCc4DFZX0xMLclvsSVm4GJkiYDRwGrbG+2/SiwCphd9r3c9k3l6mRJy7kiIqJL6rz58WWt25LmUl01jMWBtjeU826QdECJTwEebGk3WGIjxQfbxNuSNJ/qqoaDDjpojKlHRMRo6oz+2ort79L8Mypq91VjiLdle6Htftv9fX19Y0wxIiJGU2dCyWNbNvcA+hnhP+CjeFjS5HKVMpnnX/Y1CExraTcVeKjE3zUsfl2JT23TPiIiuqjOlcpftCxHAU9S9YGMxXJgXlmfB1zVEj9JlVnA4+U22UrgSEmTSgf9kcDKsu9JSbPKyLGTWs4VERFdUqdPZUzvVZF0GdVVxv6SBqlGcf09sEzSKcAvqd4iCdWQ4GOAAaohxSeX794s6Tyef9PkubaHOv8/zvNDiq8uS0REdNFIrxM+Z4TjbPu8kU5s+8Tt7HpPu5MBp23nPIuARW3ia4A3jJRDRETsXCNdqfymTewlwCnAK4ERi0pERIw/I71O+EtD65JeBpxBdVtqKfCl7R0XERHj14h9KuWJ9k8BH6Z6WPEt5SHEiIiIbYzUp/JF4FhgIfBG20/ttKwiIqInjTSk+NPAH1BN5vhQy1QtT9aZpiUiIsafkfpUdvhp+4iIGN9SOCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGdKWoSHpA0l2S1kpaU2L7SVol6b7yOanEJekiSQOS7pT0lpbzzCvt75M0rxu/JSIintfNK5V3255pu79snwVcY3sGcE3ZBjgamFGW+cDFUBUhYAFwGHAosGCoEEVERHfsSre/5gCLy/piYG5LfIkrNwMTJU0GjgJW2d5s+1FgFTB7ZycdERHP61ZRMfAjSbdJml9iB9reAFA+DyjxKcCDLccOltj24hER0SXbfUd9hx1u+yFJBwCrJP1shLZqE/MI8W1PUBWu+QAHHXTQjuYaERE1deVKxfZD5XMj8B2qPpGHy20tyufG0nwQmNZy+FTgoRHi7b5voe1+2/19fX1N/pSIiGix04uKpJdIetnQOnAkcDewHBgawTUPuKqsLwdOKqPAZgGPl9tjK4EjJU0qHfRHllhERHRJN25/HQh8R9LQ9/+r7R9KWg0sk3QK8Evg+NJ+BXAMMAD8FjgZwPZmSecBq0u7c21v3nk/IyIihtvpRcX2/cCb2sR/DbynTdzAads51yJgUdM5RkTE2OxKQ4ojIqLHpahERERjujWkuCe89TNLup3Cbu+2L57U7RQiokG5UomIiMakqERERGNSVCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjer6oSJot6V5JA5LO6nY+ERHjWU8XFUkTgK8CRwOHACdKOqS7WUVEjF89XVSAQ4EB2/fbfgZYCszpck4REeNWrxeVKcCDLduDJRYREV2wZ7cTeIHUJuZtGknzgfll8ylJ93Y0q+7aH3ik20nUpX+c1+0UdiU99bcDYEG7fwXHrZ76++mTO/y3+8M6jXq9qAwC01q2pwIPDW9keyGwcGcl1U2S1tju73YesePyt+tt+ftVev3212pghqSDJe0NnAAs73JOERHjVk9fqdh+VtLpwEpgArDI9roupxURMW71dFEBsL0CWNHtPHYh4+I2324qf7velr8fIHubfu2IiIgx6fU+lYiI2IWkqOwmMl1N75K0SNJGSXd3O5fYMZKmSbpW0npJ6ySd0e2cui23v3YDZbqa/wP8OdUw69XAibbv6WpiUYukdwJPAUtsv6Hb+UR9kiYDk23fLullwG3A3PH8716uVHYPma6mh9m+Htjc7Txix9neYPv2sv4ksJ5xPqtHisruIdPVRHSZpOnAm4FbuptJd6Wo7B5qTVcTEZ0h6aXAt4EzbT/R7Xy6KUVl91BrupqIaJ6kvagKyjdtX9ntfLotRWX3kOlqIrpAkoBLgPW2v9ztfHYFKSq7AdvPAkPT1awHlmW6mt4h6TLgJuCPJQ1KOqXbOUVthwMfAY6QtLYsx3Q7qW7KkOKIiGhMrlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYkAJP2kRpszJb24w3nMHO05B0kflfTPDX9v4+eM8SlFJQKw/Y4azc4EdqiolNcS7IiZwLh+eC56W4pKBCDpqfL5LknXSbpC0s8kfVOVTwJ/AFwr6drS9khJN0m6XdK3yqSCSHpA0jmSbgCOl/RqST+UdJuk/y3ptaXd8ZLulnSHpOvLFDvnAh8qT2Z/qEbefZK+LWl1WQ6XtEfJYWJLuwFJB7Zr3/g/zBjX9ux2AhG7oDcDr6ealPNG4HDbF0n6FPBu249I2h/4a+C9tn8j6bPAp6iKAsDvbP8pgKRrgFNt3yfpMOBrwBHAOcBRtn8laaLtZySdA/TbPr1mrl8BLrR9g6SDgJW2XyfpKuADwNfLdz5g+2FJ/zq8PfC6F/jPK+L3UlQitnWr7UEASWuB6cANw9rMAg4BbqzmFGRvqvm7hlxejn8p8A7gW6UdwD7l80bgUknLgLHObvte4JCWc7+8vIHwcqqi9XWqCUYvH6V9RCNSVCK29XTL+hba/3siYJXtE7dzjt+Uzz2Ax2zPHN7A9qnlKuJ9wFpJ27SpYQ/g7bb/31bJSTcBr5HUB8wFzh+l/Ri+OmJb6VOJqO9JYOj/6m8GDpf0GgBJL5b0R8MPKC9s+oWk40s7SXpTWX+17VtsnwM8QvVOnNbvqONHVDNUU845s3yvge8AX6aalv3XI7WPaEqKSkR9C4GrJV1rexPwUeAySXdSFZnXbue4DwOnSLoDWAfMKfEvSrpL0t3A9cAdwLVUt6dqddQDnwT6Jd0p6R7g1JZ9lwN/yfO3vkZrH/GCZer7iIhoTK5UIiKiMemoj9hFSToZOGNY+Ebbp3Ujn4g6cvsrIiIak9tfERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGN+f+UQGoL+CmGNwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(train.interest_level);\n",
    "pyplot.xlabel('interest_level');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总共有三类样本，每类样本分布的并不均匀，一般类别之间的数量差别在十倍以上时才考虑使用class_weight，这里暂不考虑"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop ids and get labels\n",
    "y_train = train['interest_level']\n",
    "\n",
    "train = train.drop([\"interest_level\"], axis=1)\n",
    "X_train = np.array(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用前几轮调好的参数：  \n",
    "n_estimator = 338  \n",
    "max_depth = 5    \n",
    "min_child_weight = 2   \n",
    "subsample = 0.8    \n",
    "colsample = 0.9  \n",
    "调节reg_alpha和reg_lambda，其他参数缺省"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'reg_alpha': [1.5, 2], 'reg_lambda': [0.5, 1, 2]}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#reg_alpha = [1e-3, 1e-2, 0.05, 0.1]    #default = 0\n",
    "#reg_lambda = [1e-3, 1e-2, 0.05, 0.1]   #default = 1\n",
    "\n",
    "reg_alpha = [ 1.5, 2]    #default = 0, 测试0.1,1，1.5，2\n",
    "reg_lambda = [0.5, 1, 2]      #default = 1，测试0.1， 0.5， 1，2\n",
    "\n",
    "param_test5_1 = dict(reg_alpha=reg_alpha, reg_lambda=reg_lambda)\n",
    "param_test5_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58061, std: 0.00355, params: {'reg_alpha': 1.5, 'reg_lambda': 0.5},\n",
       "  mean: -0.58066, std: 0.00358, params: {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       "  mean: -0.58080, std: 0.00354, params: {'reg_alpha': 1.5, 'reg_lambda': 2},\n",
       "  mean: -0.58083, std: 0.00346, params: {'reg_alpha': 2, 'reg_lambda': 0.5},\n",
       "  mean: -0.58056, std: 0.00330, params: {'reg_alpha': 2, 'reg_lambda': 1},\n",
       "  mean: -0.58127, std: 0.00337, params: {'reg_alpha': 2, 'reg_lambda': 2}],\n",
       " {'reg_alpha': 2, 'reg_lambda': 1},\n",
       " -0.5805565085180221)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb5_1 = XGBClassifier(\n",
    "        learning_rate =0.09, #上一轮尝试了0.01，跑的实在太慢了，还是设置为0.09好了\n",
    "        n_estimators=338,  #第二轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=2,\n",
    "        gamma=0,\n",
    "        subsample=0.8,\n",
    "        colsample_bytree=0.9,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch5_1 = GridSearchCV(xgb5_1, param_grid = param_test5_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch5_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch5_1.grid_scores_, gsearch5_1.best_params_,     gsearch5_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([695.16058583, 693.12254219, 692.58370132, 689.87631836,\n",
       "        704.60371871, 661.822752  ]),\n",
       " 'mean_score_time': array([1.83041172, 1.64228735, 1.51660342, 1.72414103, 1.7217392 ,\n",
       "        1.58845134]),\n",
       " 'mean_test_score': array([-0.580613  , -0.58065784, -0.58079605, -0.58082808, -0.58055651,\n",
       "        -0.58126944]),\n",
       " 'mean_train_score': array([-0.48257774, -0.48392066, -0.48660253, -0.48473995, -0.48637775,\n",
       "        -0.48825087]),\n",
       " 'param_reg_alpha': masked_array(data=[1.5, 1.5, 1.5, 2, 2, 2],\n",
       "              mask=[False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_reg_lambda': masked_array(data=[0.5, 1, 2, 0.5, 1, 2],\n",
       "              mask=[False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'reg_alpha': 1.5, 'reg_lambda': 0.5},\n",
       "  {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       "  {'reg_alpha': 1.5, 'reg_lambda': 2},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 0.5},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 1},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 2}],\n",
       " 'rank_test_score': array([2, 3, 4, 5, 1, 6]),\n",
       " 'split0_test_score': array([-0.57506134, -0.5746893 , -0.57506427, -0.57509558, -0.57540493,\n",
       "        -0.57620707]),\n",
       " 'split0_train_score': array([-0.48317104, -0.48412137, -0.48709493, -0.48557567, -0.48724063,\n",
       "        -0.48908688]),\n",
       " 'split1_test_score': array([-0.57872797, -0.57907193, -0.57946507, -0.57949373, -0.57872489,\n",
       "        -0.57906156]),\n",
       " 'split1_train_score': array([-0.48261052, -0.48393458, -0.4871958 , -0.48514942, -0.48681581,\n",
       "        -0.48850626]),\n",
       " 'split2_test_score': array([-0.58049303, -0.58110998, -0.58038808, -0.58081857, -0.58055301,\n",
       "        -0.58136191]),\n",
       " 'split2_train_score': array([-0.4823155 , -0.48421636, -0.48687655, -0.48433855, -0.48596178,\n",
       "        -0.48739681]),\n",
       " 'split3_test_score': array([-0.58414081, -0.58366311, -0.5845773 , -0.58429519, -0.58418858,\n",
       "        -0.58443486]),\n",
       " 'split3_train_score': array([-0.48304666, -0.48425537, -0.48605484, -0.48503687, -0.48654686,\n",
       "        -0.48882821]),\n",
       " 'split4_test_score': array([-0.58464307, -0.58475613, -0.58448664, -0.58443842, -0.58391215,\n",
       "        -0.58528303]),\n",
       " 'split4_train_score': array([-0.48174496, -0.48307561, -0.48579052, -0.48359927, -0.48532364,\n",
       "        -0.48743619]),\n",
       " 'std_fit_time': array([  3.26206318,   5.13731583,   4.69681835,   4.60099319,\n",
       "         28.96094864, 124.32602846]),\n",
       " 'std_score_time': array([0.51513862, 0.24251946, 0.11777716, 0.3009138 , 0.22484306,\n",
       "        0.31492927]),\n",
       " 'std_test_score': array([0.00355174, 0.00358244, 0.00354145, 0.00345535, 0.00329684,\n",
       "        0.00336566]),\n",
       " 'std_train_score': array([0.00051677, 0.00043682, 0.00057076, 0.0006952 , 0.00067071,\n",
       "        0.00070578])}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch5_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.580557 using {'reg_alpha': 2, 'reg_lambda': 1}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAELCAYAAADkyZC4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4VGX2wPHvARISSuglEJqCNEGUiBQFwQaoFEEFC0UFG2vZn7q6ui7Lumtd1y5SpFgARZQuq7tgBQWlFxEQIRCQHkIJKef3x3sTJjEhCWRyJ8n5PM88zNx5751zwfHMfe/7vkdUFWOMMaaglfI7AGOMMcWTJRhjjDFBYQnGGGNMUFiCMcYYExSWYIwxxgSFJRhjjDFBYQnGGGNMUAQ1wYhIdxH5SUQ2icij2bw/RET2iMgK73FHwHvPichaEVkvIq+IiHjb/yEi20UkMcux7hKR1d5xvhaRFsE8N2OMMacmwZpoKSKlgY3AFUAcsBQYqKrrAtoMAWJVdUSWfTsCzwOdvU1fA4+p6iIRaQ/8CvysqhUC9olS1QTveS/gHlXtHpSTM8YYk6syQTx2O2CTqm4BEJGpQG9g3Sn3chSIAMIBAcKA3QCqusQ7XuYdvOTiKe8d45SqV6+uDRs2zEM4xhhj0v3www97VbVGbu2CmWDqAtsDXscBF2XTrp+IdMZd7TyoqttVdbGILATicQnmNVVdn9sHisi9wB9xialbbu0bNmzIsmXLcj8TY4wxGUTk17y0C+Y9GMlmW9aritlAQ1VtDXwOTAIQkcZAcyAGl6i6eUnolFT1dVU9G/gT8ES2QYkMF5FlIrJsz549eT4ZY4wx+RPMBBMH1At4HQPsDGygqvtUNcl7ORZo6z3vCyxR1URVTQTmA+3z8dlTgT7ZvaGqY1Q1VlVja9TI9QrPGGPMaQpmglkKNBGRRiISDgwAZgU2EJHogJe9gPRusG1AFxEpIyJhQJeA97IlIk0CXl4N/HyG8RtjjDkDQbsHo6opIjICWACUBt5W1bUiMgpYpqqzgPu8EV8pwH5giLf7dNw9lNW4brVPVXU2uOHLwE1AORGJA8ap6khghIhcDiQDB4DBpxN3cnIycXFxHD9+/HR2N0EWERFBTEwMYWFhfodijMlF0IYpFwWxsbGa9Sb/L7/8QsWKFalWrdrvRqoZf6kq+/bt4/DhwzRq1MjvcIwpsUTkB1WNza2dzeTP4vjx45ZcQpSIUK1aNbu6NKaIsASTDUsuocv+bYwpOizBGGNMSaIK374KR/cH/aOCOdHSGGNMKElLg/kPw9JxgEDHEbnucibsCqaEW7RoEddcc80ZtzmVDRs20KFDB8qWLcsLL7yQY7shQ4bQqFEj2rRpQ5s2bVixYsVpf6YxJou0NJhzv0sune6HDvcG/SPtCibEqSqqSqlSRfe3QNWqVXnllVf45JNPcm37/PPP079//0KIypgSJC0VZt4LK6dA54eh6+NQCPczLcGcwt9mr2XdzoTcG+ZDizpR/PXalqdss3XrVnr06EHXrl1ZvHgxDzzwAKNHjyYpKYmzzz6bCRMmUKFCBebNm8cf//hHqlevzgUXXMCWLVuYM2dOtsf8/vvveeCBBzh27BiRkZFMmDCBpk2bZmozcuRINm/ezI4dO9i+fTuPPPIIw4YNAyAxMZH+/fuzZs0a2rZty7vvvouIMGrUKGbPns2xY8fo2LEjb7311u9uxNesWZOaNWsyd+7cM/ibM8acltRk+PhOWPORSyxdHim0jy66P4uLuZ9++olBgwbx2WefMX78eD7//HN+/PFHYmNjefHFFzl+/Dh33nkn8+fP5+uvvya3ddWaNWvGl19+yfLlyxk1ahR//vOfs223atUq5s6dy+LFixk1ahQ7d7rVfZYvX85LL73EunXr2LJlC9988w0AI0aMYOnSpaxZs4Zjx45lJLjRo0czevTofJ/3448/TuvWrXnwwQdJSkrKfQdjTM5STsD021xyufxvhZpcwK5gTim3K41gatCgAe3bt2fOnDmsW7eOTp06AXDixAk6dOjAhg0bOOusszImHA4cOJAxY8bkeLxDhw4xePBgfv75Z0SE5OTkbNv17t2byMhIIiMj6dq1K99//z2VK1emXbt2xMTEANCmTRu2bt3KxRdfzMKFC3nuuec4evQo+/fvp2XLllx77bXcdddd+T7np59+mtq1a3PixAmGDx/Os88+y5NPPpnv4xhjgJQk+GAwbJwPVz0NHe4p9BAswYSo8uXLA+4ezBVXXMGUKVMyvb98+fJ8He8vf/kLXbt25eOPP2br1q1ceuml2bbL2r2V/rps2bIZ20qXLk1KSgrHjx/nnnvuYdmyZdSrV4+RI0ee0STI6OjojM8aOnToKQcEGGNOIfkYTLsFNn0OV/8LLrwj932CwLrIQlz79u355ptv2LRpEwBHjx5l48aNNGvWjC1btrB161YApk2bdsrjHDp0iLp16wIwceLEHNvNnDmT48ePs2/fPhYtWsSFF16YY9v0ZFK9enUSExOZPn16Ps7s9+Lj4wGXVD/55BPOPffcMzqeMSXSiSPw/o2w6b/Q61XfkgtYggl5NWrUYOLEiQwcOJDWrVvTvn17NmzYQGRkJG+88Qbdu3fn4osvplatWlSqVCnH4zzyyCM89thjdOrUidTU1BzbtWvXjquvvpr27dvzl7/8hTp16uTYtnLlygwbNoxWrVrRp0+fTMko8B7Mrl27iImJ4cUXX+Spp54iJiaGhAQ3eKJnz54Z93luvvlmWrVqRatWrdi7dy9PPJFtSR9jTE6SDsN718PWr6DPm3DBIF/DscUusyx2uX79epo3b+5TRPmTmJhIhQoVUFXuvfdemjRpwoMPPnjaxxs5ciQVKlTgoYceKsAoC15R+jcyptAcP+SSS9wyuG4MtArecH9b7LIEGDt2LG3atKFly5YcOnSIO++80++QjDF+OHYAJveBHT/A9ROCmlzyw65givAVTHYmTJjAyy+/nGlbp06deP31132KqOAV9X8jYwrUkX3wTh/YswFumAxNewT9I/N6BWOjyIqZoUOHMnToUL/DMMYUhsQ9MLk37NsEA6ZAk8v9jigTSzDGGFMUHd4Fk3rBwW1w8wdw1qV+R/Q7lmCMMaaoObQDJl3rkswt06HhxX5HlC1LMMYYU5Qc3OaSy9H9cOvHUP8ivyPKUVBHkYlIdxH5SUQ2icij2bw/RET2iMgK73FHwHvPichaEVkvIq+IN6VcRP4hIttFJDHLsf4oIutEZJWI/FdEGgTz3IwxptDt3wITerpRY7d+EtLJBYKYYESkNPA60ANoAQwUkRbZNJ2mqm28xzhv345AJ6A1cC5wIdDFaz8baJfNcZYDsaraGpgOPFeQ51NcFUY9mPfee4/WrVvTunVrOnbsyMqVK0/7WMaUWHs3wYSr4UQiDJoFMW39jihXwewiawdsUtUtACIyFegNrMvDvgpEAOGAAGHAbgBVXeIdL/MOqgsDXi4Bbjmz8ENDcagH06hRI7744guqVKnC/PnzGT58ON99953fYRlTdPy2ASb3cnVdhsyFWv4txJsfwUwwdYHtAa/jgOyu5/qJSGdgI/Cgqm5X1cUishCIxyWY11R1fT4++3Zg/mnGfdL8R2HX6jM+TCa1W0GPZ07ZpLjVg+nYsWPG8/bt2xMXF3c6f3PGlEy71rihyKVKu+RSs5nfEeVZMH8WZ1cuLeusztlAQ69b63NgEoCINAaaAzG4RNXNS0K5f6jILUAs8HwO7w8XkWUisiy3Gip+Kq71YMaPH0+PHsGfCGZMsbBzBUy6BkqHw5B5RSq5QHCvYOKAegGvY4CdgQ1UdV/Ay7HAs97zvsASVU0EEJH5QHvgy1N9oIhcDjwOdFHVbKtVqeoYYAy4mfynPINcrjSCqTjWg1m4cCHjx4/n66+/PtO/HmOKv7gf4N2+UDYKBs+Gqo38jijfgnkFsxRoIiKNRCQcGADMCmwgItEBL3sB6d1g24AuIlJGRMJwN/hP2UUmIucDbwG9VPW3AjoH32StB7NixQpWrFjBunXrGD9+PPld4ie9HsyaNWuYPXt2jnVbTqcezPTp01m9ejXDhg3L8birVq3ijjvuYObMmVSrVi1fsRtT4mxb4rrFIqvA0HlFMrlAEBOMqqYAI4AFuOTwgaquFZFRItLLa3afNxR5JXAfMMTbPh3YDKwGVgIrVXU2ZAxfjgPKiUiciIz09nkeqAB86A15zpTMiqriUA9m27ZtXHfddbzzzjucc845p4zTmBJv69fwznVQsZbrFqtc3++ITltQJ1qq6jxgXpZtTwY8fwx4LJv9UoFslwZW1UeA3xWWVtXQWoSngATWg0mvUf/UU09xzjnnZNSDqV69Ou3aZTdy+6RHHnmEwYMH8+KLL9KtW7cc26XXg9m2bVtGPZiNGzdm2zawHkzDhg1/Vw8G4K677mLUqFHs27ePe+5xJVvLlClD1kVGjTHA5oUwZSBUaQCDZkLF2n5HdEZsNeUivJqy1YMxphj5+TOYejNUa+ySS4UafkeUI6sHUwJYPRhjiokN82DqTVCjKQyZE9LJJT/sCqYIX8Fkx+rBGFPErJsJ02+D6PPglo/cjf0QZ/VgzoCq/m40VVFR3OvBlOQfRKYYWj0dZgyHmFi4eTpERPkdUYGyLrIsIiIi2Ldvn/2PLASpKvv27SMiIsLvUIw5cyvehxnDoH4HuGVGsUsuYFcwvxMTE0NcXFyuM+ONPyIiIjImfBpTZP0wCWbfD2d1cZUow8v5HVFQWILJIiwsLGN2vDHGFLjvx8K8h6Dx5XDjuxAW6XdEQWNdZMYYU1gWv+GSS9OeMOD9Yp1cwBKMMcYUjq//DQseg+a94PpJUKZs7vsUcdZFZowxwfbFc7DwH3Buf+j7FpQuGf/rLRlnaYwxflCF/z0FX70A5w2E3q+7ui4lhCUYY4wJBlX47En49hW4YBBc8zIU4cq0p8MSjDHGFDRV+PQx+O5NuPAO6PF8iUsuYAnGGGMKVloazPs/WPY2tL8XrvoHFNGVQc6UJRhjjCkoaakw+z5Y/i5c/CBc9tcSm1zAEowxxhSM1BSYeQ+smgZd/gSXPlaikwtYgjHGmDOXmuwWrVw7A7o9AZ0f9juikGAJxhhjzkTKCZg+FDbMgSv+Dp3u8zuikGEJxhhjTlfycfhwMGz8FLo/C+3v8juikBLUcXMi0l1EfhKRTSLyaDbvDxGRPSKywnvcEfDecyKyVkTWi8gr4hVoEZF/iMh2EUnMcqzOIvKjiKSISP9gnpcxxpB8DKYOdMnl6hctuWQjaAlGREoDrwM9gBbAQBFpkU3TaaraxnuM8/btCHQCWgPnAhcCXbz2s4F22RxnGzAEeL8gz8MYY37nxBF4/wbYvBB6vQYX3u53RCEpmF1k7YBNqroFQESmAr2BdXnYV4EIIBwQIAzYDaCqS7zjZd5Bdau3Pa1AojfGmOwkHYb3boDtS9y6Yufd6HdEISuYXWR1ge0Br+O8bVn1E5FVIjJdROoBqOpiYCEQ7z0WqOr6gghKRIaLyDIRWWZFxYwx+XL8ELxzHWz/DvqNs+SSi2AmmOwGgGetQzwbaKiqrYHPgUkAItIYaA7E4JJSNxHpXBBBqeoYVY1V1dgaNWoUxCGNMSXB0f0wuTfsXA43TIJz+/kdUcgLZoKJA+oFvI4BdgY2UNV9qprkvRwLtPWe9wWWqGqiqiYC84H2QYzVGGNydmQfTO4Fu9e6KpTNr/U7oiIhmAlmKdBERBqJSDgwAJgV2EBEogNe9gLSu8G2AV1EpIyIhOFu8BdIF5kxxuRL4m8w6RrY+zMMnAJNu/sdUZERtASjqinACGABLjl8oKprRWSUiPTymt3nDUVeCdyHGwUGMB3YDKwGVgIrVXU2ZAxfjgPKiUiciIz0tl/obb8eeEtE1gbr3IwxJURCPEy8Gg5shZs+gMaX+x1RkSKqWW+LlByxsbG6bNkyv8MwxoSiQ3Ew6Vp3BXPzh9Cgo98RhQwR+UFVY3NrZzP5jTEmqwO/uuRy7ADc+jHUy27qncmNJRhjjAm0fwtMvBZOHIZBM6HuBX5HVGRZgjHGmHR7f3ZXLilJMHgORLf2O6IizRKMMcYA/LYeJvUCFIbMhVrZrWxl8qPkFYk2xpisdq12o8WkFAyZZ8mlgFiCMcaUbDuXw8RroEwEDJ0HNc7xO6JiwxKMMabkilsGk3pD2SiXXKqd7XdExYolGGNMyfTrYpjcB8pVdcmlSkO/Iyp2LMEYY0qeX76Cd/tBxdouuVSul/s+Jt8swRhjSpbN/4P3rndJZchciKrjd0TFliUYY0zJsfE/8P4Ad69lyFyoWMvviIo1SzDGmJJhw1yYehPUbAaDZ0P56n5HVOxZgjHGFH9rP4YPBkH0eTBolruxb4LOEowxpnhb9SFMvw1iLnQLV0ZW9juiEsMSjDGm+Fr+HswYBg06wc3TISLK74hKFEswxpjiadkEmHkPnHWpKxZWtoLfEZU4lmCMMcXP92NhzgPQ5EoYOBXCy/kdUYlkCcYYU7x8+xrMewiaXg03vgthEX5HVGIFNcGISHcR+UlENonIo9m8P0RE9ojICu9xR8B7z4nIWhFZLyKviIh42/8hIttFJDHLscqKyDTvs74TkYbBPDdjTAj66l/wn8ehRR+4YRKUKet3RCVa0BKMiJQGXgd6AC2AgSKS3RrY01S1jfcY5+3bEegEtAbOBS4EunjtZwPZ1S+9HTigqo2BfwPPFuT5GGNCmCosegb+Owpa3QD9xkPpML+jKvGCeQXTDtikqltU9QQwFeidx30ViADCgbJAGLAbQFWXqGp8Nvv0BiZ5z6cDl6Vf9RhjijFV+N/fYdHT0OZm6DsaSlstxVAQzARTF9ge8DrO25ZVPxFZJSLTRaQegKouBhYC8d5jgaquz+vnqWoKcAiodmanYIwJaarwnydc11jbIdDrNShV2u+ojCeYCSa7qwfN8no20FBVWwOf412BiEhjoDkQg0sc3USkcwF8HiIyXESWiciyPXv25HJIY0zIUoX5f4LFr0G74XDNS1DKxi2FkmD+a8QBgWtgxwA7Axuo6j5VTfJejgXaes/7AktUNVFVE4H5QPu8fp6IlAEqAfuzNlLVMaoaq6qxNWrUyOcpGWNCQlqaG4b8/VvQYQT0eA6sRzzk5CvBiEgpEcnrVNilQBMRaSQi4cAAYFaW40UHvOwFpHeDbQO6iEgZEQnD3eDPrYtsFjDYe94f+J+q/u4KxhhTxKWlwqwR8MNEuPiPcOVTllxCVK4JRkTeF5EoESkPrAN+EpGHc9vPuw8yAliASw4fqOpaERklIr28Zvd5Q5FXAvcBQ7zt04HNwGpgJbBSVWd78TwnInFAORGJE5GR3j7jgWoisgn4I/C7YdHGmCIuNQU+vgtWvAeXPgaXPWnJJYRJbj/yRWSFqrYRkZtxXVh/An7w7psUabGxsbps2TK/wzDG5EVqMnx0B6z7xCWWS/7P74hKLBH5QVVjc2uXly6yMK+bqg8wU1WTyebmuTHGBE1KEnw4xCWXK/9hyaWIyEuCeQvYCpQHvhSRBkBCMIMyxpgMycdh2q2wYQ70eB46jvA7IpNHuc5GUtVXgFcCNv0qIl2DF5IxxnhOHIVpN8Pm/7lhyLFD/Y7I5ENebvLf793kFxEZLyI/At0KITZjTEl24gi8fwNsXgi9X7fkUgTlpYvsNlVNAK4EagBDgWeCGpUxpmQ7ngDv9oNfv4HrxsD5t/gdkTkNeVmwJ30MYE9ggqqutDW+jDFBc+ygSy7xK6D/29Cyr98RmdOUlwTzg4j8B2gEPCYiFYG04IZljCmRju6Hd/rC7rVww2RodrXfEZkzkJcEczvQBtiiqkdFpBqum8wYYwrOkb0wuQ/s3QgD3odzrvQ7InOG8jKKLE1EYoCbvJ6xL9Jn1RtjTIE4vBsm94YDv8DAKdD4Mr8jMgUg1wQjIs/gCn695226T0Q6qupjQY3MGFMyJOyESb0gYQfc/CE0ym3hdFNU5KWLrCfQRlXTAERkErAcsARjjDkzB7fDpGtd99gtM6BBB78jMgUor6spVw54XikYgRhjSpgDW2FiT3dj/9aPLbkUQ3m5gnkaWC4iC3FDljtjVy/GmDOxb7PrFjuRCINnQp3z/Y7IBEFebvJPEZFFuPswgltN2crGGWNOz56NrlssLRmGzIHarfyOyARJXq5gUNV4AoqFicg2oH6wgjLGFFO718HkXoDAkLlQs7nfEZkgOt0rEZvJb4zJn/hVMPFqKFUGhs6z5FICnG6CsXowxpi82/Gj6xYLK+euXKo38TsiUwhy7CITkVfJPpEImUeVGWNMzrYvhXevg8jKMHgOVGngd0SmkJzqHsypaglbnWFjTO5+/Rbeux4q1ITBs6FSjN8RmUKUY4JR1UlnenAR6Q68DJQGxqnqM1neHwI8D+zwNr2mquO8954DrsZ1430G3K+qKiJtgYlAJDAvYPt5wGigAq4C581emQFjjB+2fAFTBrikMmgWREX7HZEpZEEbbiwipYHXgR5AC2CgiLTIpuk0VW3jPdKTS0egE9AaOBc3RLqL1/5NYDjQxHt097aPAx5V1VbAx8DDQTkxY0zuNv3XFQur0tDdc7HkUiIFcz5LO2CTqm5R1RPAVKB3HvdVIAIIB8oCYcBuEYkGolR1saoqMBno4+3TFPjSe/4Z0K9gTsMYky8bF7grl+pN3D2XCjX9jsj4JJgJpi6wPeB1nLctq34iskpEpotIPQBVXQwsBOK9xwJVXe/tH5fDMdcAvbzn1wP1CupEjDF5tH42TL0ZarV03WLlq/kdkfFRXlZTfiWbzYeAZao681S7ZrMt66i02cAUVU0SkbuASUA3EWkMNAfS7wh+JiKdgWOnOOZtwCsi8iRuUuiJHM5nOK6Ljfr1ba6oMQVmzQz46A6oewHc8hFE2LKFJV1ermAicAXHfvYerYGqwO0i8tIp9osj81VEDLAzsIGq7lPVJO/lWKCt97wvsERVE1U1EZgPtPeOGTgMJeOYqrpBVa9U1bbAFGBzdkGp6hhVjVXV2Bo1apz6zI0xebNyGnx0O9S7yC1cacnFkLcE0xjopqqvquqrwOW4q4u+wKlKzi0FmohIIxEJBwYQsNwMgHdPJV0vYL33fBvQRUTKiEgY7gb/em/JmsMi0l5c9bNBwEzvWDW9P0sBT+BGlBljgm35u/DxndCgE9wyHcpW9DsiEyLykmDqAuUDXpcH6qhqKpCU/S6gqinACGABLnF8oKprRWSUiKTfK7lPRNaKyErgPmCIt3067gpkNbASWBlQRfNu3IixTV6b+d72gSKyEdiAu6qZkIdzM8aciWVvw8x74eyucNMHEF4+931MiSFuMNYpGojcjrsiWMTJ5fr/ieuGGqmqRXY4cGxsrC5bZnNGjTkt370F8x+BJlfBDZMhLMLviEwhEZEfVDU2t3Z5Wa5/vIjMww07FuDPqpp+L6XIJhdjzBn45hX47C/Q7BroPwHKhPsdkQlBeVquHzfR8RLveSpZbtYbY0qQL1+A//0dWl4H142B0mF+R2RCVK73YETkGeB+YJ33uE9Eng52YMaYEKMKC592yaX1jXDdWEsu5pTycgXTE2ijqmkAIjIJWI6VTTam5FCF//4Nvv43tLkFer0CpUr7HZUJcXmdyR+4PL8NcDemJFGFBY+75BJ7G/R61ZKLyZO8XME8DSwXkYWcHEVmVy/GlARpaW6k2NKxcNFd0P0ZECtoa/ImL6PIpojIItyNfgH+RHDXMDPGhIK0NJjzAPw4CTreB1eMsuRi8iVPo8i8GfQZs/BFZBtgC3kZU1ylpcLMEbDyfbjkIej2hCUXk295Haaclf2XZkxxlZoCn9wFqz+Ero9Dl0f8jsgUUaebYE49/d8YUzSlJrtFK9fNhMtHwsUP+h2RKcJyTDAi8irZJxIh86gyY0xxkJIEHw6Fn+bCVf+EDvf6HZEp4k51BXOqRbpsAS9jipPk4/DBrfDzf6DnC9BumN8RmWIgxwSjqpOybhOR2qq6K7ghGWMK1YmjMHUgbPkCrn0Z2g7xOyJTTOR3uPG8oERR1KSlwqEdfkdhzJlLSoT3b4BfvoQ+b1hyMQUqvwnGRo8BfPMyvNEB1n7idyTGnL7jCfBuP/j1W7euWJub/I7IFDP5TTBjgxJFUXPudVC9MXw4GOb+n+u/NqYoOXYQ3ukLO5bB9ROgVX+/IzLFUL4SjKq+EaxAipQqDWHop9DxD7B0HIy7HPZu8jsqY/Lm6H6Y3AviV8IN70CL3n5HZIopW/LldJUJhyufcmViE3bAW51h1Qd+R2XMqSXugUnXwm8bYOAUaNbT74hMMWYJ5kydcxXc9TVEnwczhrn65CeO+B2VMb93eBdMugb2bYabpkGTK/yOyBRzQU0wItJdRH4SkU0i8mg27w8RkT0issJ73BHw3nMislZE1ovIKyJuISQRaSsiq71jBm5vIyJLvOMsE5F2wTy3TCrVhcGzofPDsPw9GNsNfltfaB9vTK4SdsLEq+HgdrhlOpzd1e+ITAkQtAQjIqWB14EeQAtgoIi0yKbpNFVt4z3Geft2BDoBrYFzcSs5d/HavwkMB5p4j+7e9ueAv6lqG+BJ73XhKV3GLQh468euj3tMV/hxsqulYYyfDm6HCT3h8G64dQY0vNjviEwJEcwrmHbAJlXdoqongKlAXu8mKhABhANlgTBgt4hEA1GqulhVFZgM9AnYJ8p7XgnYWTCnkU9nd3VdZvXawaw/uG6zpMO+hGIM+39xyeXofhg0E+q39zsiU4IEM8HUBbYHvI7ztmXVT0RWich0EakHoKqLgYVAvPdYoKrrvf3jcjjmA8DzIrIdeIEciqKJyHCvC23Znj17Tv/sTqViLXcl0+0JWPORGwAQvzI4n2VMTvZtdt1iJw7D4FkQ09bviEwISEtTtu49woEjJ4L+WcFMMNlNyszaXzQbaKiqrYHPgUkAItIYaA7E4BJINxHpnMsx7wYeVNV6wIPA+OyCUtUxqhqrqrE1atTI5ynlQ6nS7p7MkLlunsy4y+H7sdZlZgrHnp9gQg+3gOXgOVCnjd8RGR8kpaSyOu4Q05Zu468xHreMAAAexElEQVQz19D/zW9pNXIBl76wiPlrgr/q1+ku158XcUC9gNcxZOm2UtV9AS/HAs96z/sCS1Q1EUBE5gPtgXe842R3zMHA/d7zD4FxZ34KBaBBR9dl9sndMO8h+OUL6PUaRNqC1CZIdq+FSb3cj5whc6FmM78jMoXg0NFk1sYfYt3OBPeIT2DTb4mkpLkfteXDS9OiThT928bQsk4lOpxdLegxBTPBLAWaiEgjYAcwAMi0FoWIRHvVMgF6AelDr7YBw0TkadxVSxfgJVWNF5HDItIe+A4YBLzq7bPTa7cI6Ab8HKwTy7fy1WDgVFjyOnw+Et66BPpPgJhYvyMzxU38SpjcB8pEuJGN1Rv7HZEpYKrKjoPHWLczgbVeIlm3M4EdB49ltKkVVZYW0VFc1rwmLetUokV0FPWrlqNUqcJd7StoCUZVU0RkBLAAKA28raprRWQUsExVZwH3iUgvIAXYDwzxdp+OSxKrcV1gn6rqbO+9u4GJQCQw33sADANeFpEywHHcSLPQUaqUm/lfv4OrufH2VXDZX6HDCPeeMWdqxw9u+ZeyUe6eS9Wz/I7InKHk1DQ270k8mUy8hHLoWDLgqlifVb08FzSowq0dGtAiOooWdaKoXqGsz5E7oiX4nkBsbKwuW+ZDaZtjB9wIs/WzocmV0Ge0u8ox5nRt+w7e6w+RVWDIHKhc3++ITD4lJqWwIT5zIvlp92FOpKQBULZMKZpFR9GyTlRGImlWuyLlwoPZEZU9EflBVXPtgrEE40eCAXezf+k4WPBnKFcd+o2Dhp38icUUbVu/cUvuV6jlusUqZTdY04QKVWXP4aRM3Vtrdx5i676jGW2qlg/PlEha1omiYbXylCkdGr0deU0whZ/6jCPiqgbWawcfDnFLeFz6Z7jkj+7mrDF5sWURTBkIleq5brGKtf2OyARITVN+2XskUyJZH5/A3sSTQ4TrVy1HyzpR9LsghpZ1o2gRXYlaUWXxFikp0izB+C36PLjzS5j9ACx8Cn79GvqOcXNpjDmVTZ/D1Juh6tluEmWFIA67N7k6diKVn3Yfzkgk6+IT2BB/mGPJqQCElRbOqVWRrk1relcllWgWXZGoiDCfIw8eSzChoGxF10V2VheY9wiMvhiuG2PrRZmc/TQfPhgENZrCrTPtHl4h23/khHef5FDGPZPNexLxRgRTMaIMLaKjGNCuXsYorsY1KxBeJjS6uAqLJZhQIQIXDIK6sTB9qBsN1Pkh6PKoW+fMmHTrZrn/Rmq3dmuLRVbxO6JiS1XZvv9YpkSyLj6B+EMniwzWqRRBizpR9GgVTQvvJnxMlchi0cV1puz/XKGmVgsY9j+Y/wh8+by7gdtvnN24Nc6aj+CjYW4O1c0fQkQlvyMqNk6kpPHzb4czJZL1OxM4nJQCQOlSwtk1ytP+rGoZN99bREdRpXy4z5GHLkswoSi8PPR+HRp2hjkPui6zvqNd7RlTcq2c6laEqN/B1XMpW9HviIqshOPJmWa8r92ZwKbfDpOc6vq4IsNK0zy6In3Or5uRSJrWrkhEmA3AyQ9LMKHsvBuhbls3yuz9G9xEzW5PumqapmT58R03d6pRZ1eJMry83xEVCapK/KHjAYnE3Xzfvv/krPfqFcrSsk4UlzatkdHF1aBaeUoX8qz34sgSTKir3hju+Bz+8zh8+yr8uhj6vw1VGvgdmSksS8fB3P+DxpfDje9CWKTfEYWklNQ0tuw9kmkU17qdCRw4enLWe6Nq5WkdU5kBF9Z380zqRFGzYoTPkRdflmCKgrAIuPpf0PAS9yt29CXQ+zVo0cvvyEywLXkTPn0UzukBN0yCMqGxBIjfjiSlsGHXYS+JuAUeN+w6TJI36z28TCma1a7IVS1rZySSprWjqFDW/pdXmOxvuyhp2cfNm5l+G3xwK7QbDlf83SUgU/x88zJ89iQ07wX9xpfYrtE9h5NOdm95XV2/7D2SUfmiUmQYLetEcWv7BhkTFc+qUZ6wEJn1XpJZgilqqjaC2xbAf/8Gi1+DbUvg+olQ7Wy/IzMF6Yvn3cTbc/u5ibclYKh6Wpry6/6jmRLJ2p0J7DmclNEmpkokLaKj6H2ed/O9ThR1KkXYkOAQVfz/qy2OyoTDVf9wtdU/udtVzLzmJWh9vd+RmTOlCgv/CV8+B60HQJ83iuXSQceTU9nozXpPTyTr4xM4esLNei9TSmhcswKdm9TIGMXVIjqKSuWK76z34sgSTFHWtIcrZjb9dphxhytm1uM5CC/nd2TmdKjC5391XWPn3wrXvlwsksvBoycyJZJ1OxPYtCeRVG/ae4Wybtb7DbH1MuaXNKlVgbJliv65l3SWYIq6SjGuauGif8JXL0LcMrh+AtRs7ndkJj9U3craS96A2Nuh5wtFrk6QqhJ34FjAwo7uqiSwEFbtKDfr/YoWtTJuvterUviFsEzhsOX6/VquPxg2/Rc+vhOSEqHn83D+LW5spgltaWkw/2E3HLn9PXDVP0P+3y05NY1NvwUUwvJK9SYcd7PeSwmcVaNCpuXmm0eHTiEsc2Zsuf6SqPFlrstsxjCYNQJ++RKuedFmfIeytDSYcz/8OBk63Q+X/y3kksvh48ls2HWYtTu8uSXxCWzclciJVDckOCKsFM1qR3HNeXUyapg0qx1FZLh1cZV0lmCKm4q14dZP4Kt/waKnXRnd6ydCdGu/IzNZpaXCzHth5RTo/Ah0/bOvyUVV+e1w0u9Gcf2aTSGsoRc3zJj13qh6BZv1brIV1AQjIt2Bl4HSwDhVfSbL+0OA54Ed3qbXVHWc995zwNVAKeAz4H5VVRFpC0wEIoF5AdunAU2941QGDqpqmyCeXugqVRq6PAINOsJHd8C4y92oswvvCLlfxyVWarLrzlzzEXR9Aro8XLgfn6b8sjcxU1XFdTsT2HfkZCGsBtXK0SI6iuvbxngjuYpPISxTOIKWYESkNPA6cAUQBywVkVmqui5L02mqOiLLvh2BTkD6z+6vgS7AIuBNYDiwBJdgugPzVfXGgP3/BRwq6HMqchpe7LrMPr4L5j0EW7+Ca1+ByMp+R1aypZyAj26H9bPgilGuayyIjp1IZcOuzKO4NuxK4Hiy6+JKL4R1WfOa3j2TSjSPrkjFYlwIyxSOYF7BtAM2qeoWABGZCvQGsiaY7CgQAYQDAoQBu0UkGohS1cXeMScDfYD56TuK+3l1A9Ct4E6lCCtfHW76ABa/Cv8dBTtXuFFmddv6HVnJlJIEHwyGjfOh+zPQ/u4CPfy+xKRMo7jWxSewJUshrJZ1oripXYOMm+9n1yh5hbBM4QhmgqkLbA94HQdclE27fiLSGdgIPKiq21V1sYgsBOJxCeY1VV0vIrHecQKPmbVQyiXAblX9uaBOpMgrVcr9Sq7fwS0zM/4quOJvbsSSdXcUnuRjMO0WV+r46n+5LsvTlJambD9wNFMiWbczgV0JJwth1a0cSfPoKHq2is64+W6FsExhCmaCye6/4qxjomcDU1Q1SUTuAiYB3USkMdAciPHafeYloWP8XtZjDgSm5BiUyHBcFxv169fP9SSKlXrt4K6vYOYIN+fily+hz5tQrqrfkRV/J47AlIHu77zXq656aR4lpaTy8+7EjBvv6X8mBhTCalyjAh3OrpZx4725FcIyISCYCSYOqBfwOgbYGdhAVfcFvBwLPOs97wssUdVEABGZD7QH3uFk0vndMUWkDHAdkGP/j6qOAcaAmweTrzMqDiKruCXfvx8D/3nCFTPrNx4adPA7suIr6TC8fyNsW+wKx503IMemh44lZ0oka3ceYtNviaR4fVzlwkvTPDqKvufXzZioeE4tK4RlQlMwE8xSoImINMKNEhsA3BTYQESiVTXee9kLWO893wYME5GncVdCXYCXVDVeRA6LSHvgO2AQ8GrAIS8HNqhqYDeayUoELrrTXdF8OBQmXg3dHodODxa52eMh7/gheO96t8JCv3Fu8UrckOCd6YWwAuqXxB04eZFeo2JZWkRH0a1ZzYz1uBpWK2+z3k2REbQEo6opIjICWIAbpvy2qq4VkVHAMlWdBdwnIr2AFGA/MMTbfTruJv1qXBfYp6o623vvbk4OU55PwA1+XBLLsXvMZFHnfLjzS5jzgBsA8MtXcN0YqFDT78iKh2MH4J3r0F2r2HnFm3yffBHr5q7LuGdyMEshrDb1KnPTRfUzZr9bISxT1NlSMcVpqZjTpQo/ToL5f4KISi7JnHWp31EVSa4QVgKbt/5Kp8XDqHn8F0akPsiC5POBk4Ww0m+6t6hTiWa1K1LeCmGZIsSWijF5JwJth0DMhfDhEJjcBzo/DF3+VCLqkJyu3w4fzzSKa/3OBH7Zd4Sqeoh3w/9JtVK7eLH6SOo3uox/14miZZ1KnFW9PGWsEJYpIewKxq5gMjtxBOY9DCvegwad3H2DqDp+R+WrtDRl674jmYYDr92ZwN7Ek4Ww6lV1hbBiqyUzcMO9lDu6Exk4FTn7Uv8CNyZI7ArGnJ7w8q7IVcNLYO7/uVFmfUbDOVf6HVmhOJ6cyk8Ztd7dzfcNuw5nKoTVpFZFupxTI2MUV/PoKCpFhsGhHTDpWji+G275CBp28vlsjPGXJRiTvTYD3Wz/6UPh/euh431w2ZNQuvgsH3LgyIlMiWRdfAKb9xzJvhCWd88kx0JYB7e55HJ0P9wyA+pnN6fYmJLFEozJWY1z4I7P3aTMb19x8zj6jYcqDfyOLF/SC2Gd7OJyqwXvPHRy1nt6IayrWtbOGMWV50JY+7fApF6QlACDPrFleIzxWIIxpxYWCdf823WZzb4f3roEer8Oza/1O7JsnUjxCmHFJ2Radv5wlkJYFzaqmpFIWkRHUe10C2Ht3eSuXFKOw+DZEH1eAZ6NMUWbJRiTN+deB3XauImZ026BdnfClX+HMoVboVBVOXA0mbgDR9lx4BhxB46x46D7M+7AUbbsOZKpEFbz6Ch6nVfHW9ixEk1rVSy4Qli/bYDJvUDTYMgcqNWyYI5rTDFhCcbkXdWz4Pb/wOcjXe347Uug/wSodnaBfURamrI3MYntGYnDJZL0JLLjwDGOJadm2qdC2TLUrRxJ3SqRdGlaw1uPqxKNqpcPXiGsXWtgcm8oVQaGzIUaTXPfx5gSxoYp2zDl07NhHnxyt6vKeO1L0Kp/nnZLSU1j9+Ek4vYfzZQ00pPJzoPHM65A0lUuF0ZMlUiXRCqXc8+91/WqlCMqskzhrhC8cwW80wfCyrlusQJMsMYUBTZM2QRXs56umNlHt7vHL19Cj2c5IWWJP3QyccQFXIXEHTjGroTjGaO00lWvUJa6VSJpWbcSV7WsHZBAylG3SiQVQmmWe9wP8G5fKFsJBs+Cqo38jsiYkBVC31xTFBw7kXqy6+pgGjujX6TNkde54sdJbFq+kLuS/sCmtJMlekTcCK26lSO5sGEV6laJJKZKuYwurbqVI4vOSsDblsC7/aF8NXflUrmElXswJp8swZhMDh9Pdglkv+u22pHlCiSwZju4iYfRla9nWdWW/OHQ88yP+AvLW/2FlNYDiKlcjtqVIopHtcStX8N7N0BUtEsuJXx1A2PywhJMCaKqHDyanJE04rLcPI87cJQEbzhvuvAypYjxrjaurBNF3creFYh39VErKsK7kd4NEvrBjGG0W/k46GpXtbE4JJfNC12xsCoNYNAsqFjL74iMKRIswRQjqsqexKRMw3fTE0d6Iklf8iRd+fDSGd1WbRukd2FFZnRhVS9fNu/1R6KiYdBM+PJ5WPQM7PgBrp8AtVsF4WwLyc+fwdSboVpjd24VavgdkTFFhiWYIiQ1TdmdcDzH4btxB49xIiXzCKxKkWHUrRxJw2rl6dS4esYVSHoSqVwurGBHYJUqDZc+Cg06wkfDYOxl0P1piL3N3ZApSjbMgw8HQ83mcOsnVlramHyyBBNCklPTiD94nLiDR383fHfHwWPEHzyeUTo3XfUK4dStHEnz6Cgub1HLSyAnb6BXjPBp7bBGnd0os4/vhLl/9GrRv+LqzRQF62bC9NvczPxbZkBkZb8jMqbIsQRTiI4npwZ0Wx1jx8HMs9F3JRwncFqSCNSqGEHdKpFcUL8KdVtHZh6FVTmy4GalB0OFGnDzdPj2Zfjv3yF+BfR/O/TX6lo9HWYMd/Vxbv4QIqL8jsiYIskSTAFKTErxrjqOZpkH4p4H1g8BKF1KiK7khvB2PLu6Sx4BVyDRlSKL/gisUqXg4gehfkd3RTD+KrhiFLS/OzS7zFa8DzPvdfHeNA3KVvA7ImOKLEswp2F9fALfbt6XOZkcPJZRYz1deJlSGVcalzevmXHjPH0UVq2KZUtOdcP6F8FdX7n/eS94DLZ+5RbNDKX7Gj9Mcgt6ntUFBkyB8HJ+R2RMkRbUBCMi3YGXgdLAOFV9Jsv7Q4DngR3eptdUdZz33nPA1UAp4DPgflVVEWkLTAQigXnp2719/gCMAFKAuar6SDDO65tNe3lq7nrKhZfOSBrn16+cMfM8xrsSqV4hHyOwSoJyVWHA+/DdaPjPX2D0JdB/PNRv73dk8P1YmPcQNL4CbnwXwiL8jsiYIi9oCUZESgOvA1cAccBSEZmlquuyNJ2mqiOy7NsR6AS09jZ9DXQBFgFvAsOBJbgE0x2YLyJdgd5Aa1VNEpGaQTkx4PrYelx3QQxVCnoEVkkg4rrH6l3kiplN6AndnoBOD7juND8sfsNdVTW92g2rLuQVoo0proL5jW4HbFLVLap6ApiKSwB5oUAEEA6UBcKA3SISDUSp6mLvqmUy0Mfb527gGVVNAlDV3wruVDKrFBlG1fLhllzORN0L4M4voUUv+O/f4L1+kLin8OP4+t8uubToDTdMsuRiTAEKZoKpC2wPeB3nbcuqn4isEpHpIlIPQFUXAwuBeO+xQFXXe/vH5XDMc4BLROQ7EflCRC4s2NMxBS6iklvu/5p/w9ZvYHQnN5y5sHzxnCs90Op66Pd2sSoHbUwoCGaCye7nfdbaALOBhqraGvgcmAQgIo2B5kAMLoF0E5HOuRyzDFAFaA88DHwg2VxiiMhwEVkmIsv27PHhF7PJTMRNwhz2Pygb5UoPL3zalQEIFlU3bHrhP+C8m6DvW1DaxrsYU9CCmWDigHoBr2OAnYENVHVfepcWMBZInyDRF1iiqomqmgjMxyWOOO842R0zDpihzvdAGlA9a1CqOkZVY1U1tkYNW/YjZNQ+F4YvgvMGwBfPuGJeCfEF/zmq8NmT8NULcMFgN5KtVAjPJTKmCAtmglkKNBGRRiISDgwAZgU28O6ppOsFrPeebwO6iEgZEQnD3eBfr6rxwGERae9dnQwCZnr7fAJ08457Du7+zd7gnJoJirIVoO9o6POmW8dsdCf4+fOCO74qfPoYfPsKXDgMrnnJv4EFxpQAQft2qWoKbsjwAlzi+EBV14rIKBHp5TW7T0TWishK4D5giLd9OrAZWA2sBFaq6mzvvbuBccAmr818b/vbwFkisgY3oGBw+vBlU8S0ucldzVSo7W7+f/ZXSE3Oba9TS0tzS9Z89ya0vxd6Pm/JxZggs5LJVjI5dCUfg08fhR8mQkw7t8xM5Xq57vY7aakw+z5Y/q5bVeCyv4bmKgLGFBF5LZlsP+FM6AqLhGtfhn7j4bf1MPpi2DA3f8dITYFP7nbJpcujllyMKUSWYEzoa9Uf7vzCFfyaehPMfxRSknLfLzUZZgyDVdOg21+g62OWXIwpRJZgTNFQ7Wy4/TO46C53H2X8lbB/S87tU07Ah0Ng7Qy48ino/FChhWqMcSzBmKKjTFno8Szc+B4c+AVGd4Y1M37fLvk4fHArbJgDPZ6Djn8o/FiNMZZgTBHU/BpXzKxmM7ee2ewH3IAAcH9OHQgbP3UrBFx0p7+xGlOCWYIxRVPl+jB0PnS6H36Y4Eoz71wO798Amxe6CZSxt/kdpTElmq2PYYqu0mGueFnDS1xp5jGXgpRyS7+cd6Pf0RlT4lmCMUVfkytcl9nnI6H5te5hjPGdJRhTPETVgevG+B2FMSaA3YMxxhgTFJZgjDHGBIUlGGOMMUFhCcYYY0xQWIIxxhgTFJZgjDHGBIUlGGOMMUFhCcYYY0xQlOiKliKyB/jV7zgCVAf2+h3EKYR6fBD6MYZ6fBD6MYZ6fFD8Y2ygqjVya1SiE0yoEZFleSlD6pdQjw9CP8ZQjw9CP8ZQjw8sxnTWRWaMMSYoLMEYY4wJCkswoSXUV2sM9fgg9GMM9fgg9GMM9fjAYgTsHowxxpggsSsYY4wxQWEJppCJSHcR+UlENonIozm0uUFE1onIWhF5P9RiFJH6IrJQRJaLyCoR6VnI8b0tIr+JyJoc3hcRecWLf5WIXBBi8d3sxbVKRL4VkfMKM768xBjQ7kIRSRWR/oUVW8Bn5xqjiFwqIiu878oXoRSfiFQSkdkistKLb2ghx1fP+56u9z7//mzaBPe7oqr2KKQHUBrYDJwFhAMrgRZZ2jQBlgNVvNc1QzDGMcDd3vMWwNZCjrEzcAGwJof3ewLzAQHaA9+FWHwdA/59exR2fHmJMeC/hf8B84D+oRYjUBlYB9T3Xhf2dyW3+P4MPOs9rwHsB8ILMb5o4ALveUVgYzbf5aB+V+wKpnC1Azap6hZVPQFMBXpnaTMMeF1VDwCo6m8hGKMCUd7zSsDOQowPVf0S92XNSW9gsjpLgMoiEl040eUen6p+m/7vCywBYgolsMwx5PZ3CPAH4COgsP8bBPIU403ADFXd5rUv1DjzEJ8CFUVEgApe25TCiA1AVeNV9Ufv+WFgPVA3S7OgflcswRSuusD2gNdx/P4f/BzgHBH5RkSWiEj3QovOyUuMI4FbRCQO9+v2D4UTWp7l5RxCxe24X5AhRUTqAn2B0X7HcgrnAFVEZJGI/CAig/wOKIvXgOa4H2CrgftVNc2PQESkIXA+8F2Wt4L6XSlTUAcyeSLZbMs6jK8MrpvsUtwv269E5FxVPRjk2NLlJcaBwERV/ZeIdADe8WL05cuTjbycg+9EpCsuwVzsdyzZeAn4k6qmuh/gIakM0Ba4DIgEFovIElXd6G9YGa4CVgDdgLOBz0TkK1VNKMwgRKQC7kr0gWw+O6jfFUswhSsOqBfwOobfdy/FAUtUNRn4RUR+wiWcpYUTYp5ivB3oDqCqi0UkAreukS9dKdnIyzn4SkRaA+OAHqq6z+94shELTPWSS3Wgp4ikqOon/oaVSRywV1WPAEdE5EvgPNy9hlAwFHhG3c2OTSLyC9AM+L6wAhCRMFxyeU9VZ2TTJKjfFesiK1xLgSYi0khEwoEBwKwsbT4BugKISHVcN8CWEItxG+5XIyLSHIgA9hRijLmZBQzyRsi0Bw6parzfQaUTkfrADODWEPq1nYmqNlLVhqraEJgO3BNiyQVgJnCJiJQRkXLARbj7DKEi8HtSC2hKIX6XvXs/44H1qvpiDs2C+l2xK5hCpKopIjICWIAbofO2qq4VkVHAMlWd5b13pYisA1KBhwvzF24eY/w/YKyIPIi7nB7i/UorFCIyBdeFWN27D/RXIMyLfzTuvlBPYBNwFPdLstDkIb4ngWrAG94VQooW8sKIeYjRd7nFqKrrReRTYBWQBoxT1VMOuy7M+IC/AxNFZDWuK+pPqlqYKyx3Am4FVovICm/bn4H6ATEG9btiM/mNMcYEhXWRGWOMCQpLMMYYY4LCEowxxpigsARjjDEmKCzBGGOMCQpLMMYYY4LCEowxRYC3LP2cM21jTGGyBGNMAfBmQtv3yZgA9oUw5jSJSEOvmNMbwI/ArSKyWER+FJEPvUUGEZGeIrJBRL72ijvleJUhIu3EFSFb7v3ZNJs2I0XkHRH5n4j8LCLDAt6uICLTvc97z1suBBF5UkSWisgaERmTvt2YYLIEY8yZaQpMBq7ALQJ6uapeACwD/ugtBPoWblHLi3GFp05lA9BZVc/HLSnzzxzatQauBjoAT4pIHW/7+cADuEJwZ+GWCwF4TVUvVNVzcSsPX5PvMzUmn2wtMmPOzK+qukRErsH9T/0b7+IgHFiMWz13i6r+4rWfAgw/xfEqAZNEpAlunbewHNrNVNVjwDERWYgrFHcQ+F5V4wC89acaAl8DXUXkEaAcUBVYC8w+vVM2Jm8swRhzZo54fwrwmaoODHxTRM7P5/H+DixU1b5ekahFObTLuohg+uukgG2pQBnvKuoNIFZVt4vISNwK2MYElXWRGVMwlgCdRKQxgIiUE5FzcF1eZ3nJAuDGXI5TCdjhPR9yina9RSRCRKrhVvQ9Vb2g9GSy17sv1D+XGIwpEJZgjCkAqroHlxCmiMgqXMJp5nVj3QN8KiJfA7uBQ6c41HPA0yLyDa5cQk6+B+Z6n/N3Vc2xSJRXDXUsrmzvJxRe8TpTwtly/cYEmYhUUNVEb+TW68DPqvrvMzjeSCBRVV8oqBiNCQa7gjEm+IZ5N9zX4rrA3vI5HmMKhV3BGOMDERkK3J9l8zeqeq8f8Zj/b8eOaQAAAAAE9W9tCj8o4SYHgQFgYZEBsBAYABYCA8BCYABYCAwAiwCyMg4MPyAnaAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch5_1.best_score_, gsearch5_1.best_params_))\n",
    "test_means = gsearch5_1.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch5_1.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch5_1.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch5_1.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch5_1.cv_results_).to_csv('my_preds_reg_alpha_reg_lambda_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(reg_alpha), len(reg_lambda))\n",
    "train_scores = np.array(train_means).reshape(len(reg_alpha), len(reg_lambda))\n",
    "\n",
    "#log_reg_alpha = [0,0,0,0]\n",
    "#for index in range(len(reg_alpha)):\n",
    "#   log_reg_alpha[index] = math.log10(reg_alpha[index])\n",
    "    \n",
    "for i, value in enumerate(reg_alpha):\n",
    "    pyplot.plot(reg_lambda, -test_scores[i], label= 'reg_alpha:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'reg_alpha' )                                                                                                      \n",
    "pyplot.ylabel( '-Log Loss' )\n",
    "pyplot.savefig( 'reg_alpha_vs_reg_lambda1.png' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最佳参数reg_alpha = 2, reg_lambda = 1，最佳调优结果logloss = 0.580557，是调优所有轮数里面得到最好的一个结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 每轮参数及结果汇总"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "      learning_rate  n_estimators   TreeDepth   ChildWeight   subsample   colsample   alpha   lambda     logloss\n",
    "\n",
    "round1：    0.1        --338--        5          1       0.5        0.8    default   default    0.586325\n",
    "round2：    0.1         338        --6--       --3--      0.5        0.8    default   default    0.5840985\n",
    "round3：    0.1         338        --5--       --2--      0.5        0.8    default   default    0.583831 \n",
    "round4：    0.1        --276--        5          2       0.5        0.8    default   default    0.587131\n",
    "round5：    0.09       --338--        5          2       0.5        0.8    default   default    0.586436\n",
    "round6：    0.01       --1000--       5          2       0.5        0.8    default   default    0.598229\n",
    "round7：    0.09        338          5          2       0.8        0.9    default   default    0.581140\n",
    "round8：    0.09        338          5          2       0.8        0.9      2       1      0.580557"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "参数调优是个很繁琐的过程，但是掌握一定的规律就可以使调优过程更顺利，先粗调再细调，事实上最理想的办法应该是几个参数同时调，但本次作业参数太多了，而且训练模型的时间很长，实际应用中数据集更大，应该也无法进行所有参数同时调，所以日后还是要多总结调参数的技巧，虽然每次调单个参数可能达不到最优解，但训练相对简单，也能降低计算量（Ps.再来几次这样的训练作业，估计我用来训练模型的小霸王学习机要提前退休了）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
